On 5/19/16 10:43 AM, Steven Schveighoffer wrote:

> Or even better:
>
> template(T) foo if (someConstraints)
> {
>     struct Result
>     {
>        T t;
>     }
>
>     auto foo(T t)
>     {
>        return Result(t);
>     }
> }

This solution works awesomely, actually. It even produces smaller code than moving the struct completely outside.

I'm going to use this mechanism in iopipe to get my voldemorts back :)

-Steve

On Thursday, 19 May 2016 at 12:38:09 UTC, Steven Schveighoffer wrote:
> On 5/17/16 6:04 PM, ZombineDev wrote:
>> On Tuesday, 17 May 2016 at 21:58:06 UTC, Andrei Alexandrescu wrote:
>>> On 05/17/2016 05:44 PM, Georgi D wrote:
>>>> Hi,
>>>>
>>>> While working on a D project which heavily uses the lazy algorithms for
>>>> ranges I noticed a sudden huge increase in the compilation time and the
>>>> produced binary size.
>>> [snip]
>>>
>>> Thanks. That's definitely deserving of a bug report. -- Andrei
>>
>> I'm guessing that is highly related to this one:
>> https://issues.dlang.org/show_bug.cgi?id=15831
>
> Yep. chain uses voldemort type, map does not.
>
> Georgi: try copying chain implementation into your local file, but instead of having a static internal struct, move it to a module-level struct (you probably have to repeat the template parameters, but not the constraints). See if it produces a similar slimming effect.
>
> -Steve

Yes,

Making a local copy of chain and moving the structure outside of the method solved the problem and reduced the code size.

The stripped size even reduced from the version that did not experience the huge increase. Stripped: 7.4Mb -> 5.5MB

Should we actually change the implementation in phobos to be like this?

In the company I work in we are very sensitive to the code size so anything that can be made for this in phobos will help with adoption.

I am still curious though why the code size increased so dramatically with such a small change. Both versions return a result from chain and actually the one that does not experience the increase is more complex (includes the one that does).

In my actual code I have multiple calls to chain on top of the result of this and the code size does not increase dramatically.

I am also concerned by the fact that ldc on mac just could not compile the source which produced the big binary (entering an infinite loop).

Thanks

On 5/19/16 11:56 AM, Georgi D wrote:
> On Thursday, 19 May 2016 at 12:38:09 UTC, Steven Schveighoffer wrote:
>> On 5/17/16 6:04 PM, ZombineDev wrote:
>>> I'm guessing that is highly related to this one:
>>> https://issues.dlang.org/show_bug.cgi?id=15831
>>
>> Yep. chain uses voldemort type, map does not.
>>
>> Georgi: try copying chain implementation into your local file, but
>> instead of having a static internal struct, move it to a module-level
>> struct (you probably have to repeat the template parameters, but not
>> the constraints). See if it produces a similar slimming effect.
>>
>
> Yes,
>
> Making a local copy of chain and moving the structure outside of the
> method solved the problem and reduced the code size.

Thanks, that confirms the bug reports are for the same issue.

>
> The stripped size even reduced from the version that did not experience
> the huge increase. Stripped: 7.4Mb -> 5.5MB
>
> Should we actually change the implementation in phobos to be like this?

I'd much prefer we fix the compiler to handle voldemort types in a more sane way rather than try and fix the code to deal with compiler deficiencies. However, if this isn't something that's easily done (or done within a decent time frame), we may look at that solution.

> In the company I work in we are very sensitive to the code size so
> anything that can be made for this in phobos will help with adoption.
>
> I am still curious though why the code size increased so dramatically
> with such a small change. Both versions return a result from chain and
> actually the one that does not experience the increase is more complex
> (includes the one that does).

Each call to a voldemort wrapping function (one that wraps one type into another "voldemort" type, or internal struct) increases the symbol size at a rate of 3^n. Non-voldemort types do not suffer from this, because the symbol size increases at just n.

> In my actual code I have multiple calls to chain on top of the result of
> this and the code size does not increase dramatically.
>
> I am also concerned by the fact that ldc on mac just could not compile
> the source which produced the big binary (entering an infinite loop).

I would expect the possibility that the issue is with the linker. In my toy experiments, the compiler works just fine and produces an object file. It's the linker having issues with parsing those humungous symbols.

-Steve

On 05/19/2016 11:56 AM, Georgi D wrote:
> Making a local copy of chain and moving the structure outside of the
> method solved the problem and reduced the code size.
>
> The stripped size even reduced from the version that did not experience
> the huge increase. Stripped: 7.4Mb -> 5.5MB

Thanks very much for helping with these measurements! -- Andrei

On 5/19/2016 6:45 AM, Andrei Alexandrescu wrote:
> I very much advocate slapping a 64-long random string for all Voldermort returns
> and calling it a day. I bet Liran's code will get a lot quicker to build and
> smaller to boot.

Let's see how far we get with compression first.

  https://github.com/dlang/dmd/pull/5793

Using 64 character random strings will make symbolic debugging unpleasant.

On 5/19/2016 8:39 AM, Steven Schveighoffer wrote:
>> template(T) foo if (someConstraints)
>> {
>>     struct Result
>>     {
>>        T t;
>>     }
>>
>>     auto foo(T t)
>>     {
>>        return Result(t);
>>     }
>> }
>
> This solution works awesomely, actually. It even produces smaller code than
> moving the struct completely outside.
>
> I'm going to use this mechanism in iopipe to get my voldemorts back :)


Consider using a 'static struct'. A non-static struct will include a hidden member that points to the stack frame of foo(), i.e. "produces smaller code".

On Thursday, 19 May 2016 at 22:16:03 UTC, Walter Bright wrote:
> Using 64 character random strings will make symbolic debugging unpleasant.

Using 6.4 megabyte strings already makes symbolic debugging unpleasant.

The one thing that worries me about random strings is that it needs to be the same across all builds, or you'll get random linking errors when doing package-at-a-time or whatever (dmd already has some problems like this!). But building a gigantic string then compressing or hashing it still sucks... what we need is a O(1) solution that is still unique and repeatable.

On Thursday, 19 May 2016 at 22:17:37 UTC, Walter Bright wrote:
>
> Consider using a 'static struct'. A non-static struct will include a hidden member that points to the stack frame of foo(), i.e. "produces smaller code".

Queue me going to the language reference to look up static structs and reading

"A struct can be prevented from being nested by using the static attribute, but then of course it will not be able to access variables from its enclosing scope."

Not a fan of that description...

On 5/19/2016 4:36 PM, jmh530 wrote:
> Queue me going to the language reference to look up static structs and reading
>
> "A struct can be prevented from being nested by using the static attribute, but
> then of course it will not be able to access variables from its enclosing scope."
>
> Not a fan of that description...

You can always submit a PR for a better one.

On Thursday, 19 May 2016 at 22:46:02 UTC, Adam D. Ruppe wrote:
> On Thursday, 19 May 2016 at 22:16:03 UTC, Walter Bright wrote:
>> Using 64 character random strings will make symbolic debugging unpleasant.
>
> Using 6.4 megabyte strings already makes symbolic debugging unpleasant.
>
> The one thing that worries me about random strings is that it needs to be the same across all builds, or you'll get random linking errors when doing package-at-a-time or whatever (dmd already has some problems like this!). But building a gigantic string then compressing or hashing it still sucks... what we need is a O(1) solution that is still unique and repeatable.

Clearly the reason for building such a gigantic string was some sort of repetition. Detect the repetition on-the-fly to avoid processing all of it. This way the generated name is already compressed.