I have a large named enum (currently 645 members) of IRC event types. It's big by neccessity[1].

I'm using dub, and both dmd and ldc successfully build it in test and debug modes, but choke and die on plain and release. I bisected it down to when I did a big addition to the enum to encompass virtually all event types there are.

dmd -v:
[...]
> code      common
> function  kameloso.common.Separator.__xopEquals
> function  kameloso.common.Separator.__xtoHash
> function  kameloso.common.Settings.__xopEquals
> function  kameloso.common.Settings.__xtoHash
> function  kameloso.common.scopeguard
> function  kameloso.common.scopeguard.scopeString
> function  kameloso.common.scopeguard.entryString
> function  kameloso.common.KamelosoLogger.this
> function  kameloso.common.KamelosoLogger.writeLogMsg
> function  kameloso.common.KamelosoLogger.beginLogMsg
> function  kameloso.common.KamelosoLogger.logMsgPart
> function  kameloso.common.KamelosoLogger.finishLogMsg
> zsh: segmentation fault (core dumped)  dmd -c -v  -w -d -version=Have_kameloso -Isource/  source/arsd/dom.d
Where it stops here varies if I comment stuff out, so I don't think the Logger is at fault.

ldc -v:
[...]
> code      irc
> code      constants
> code      connection
> code      config
> code      common
> /usr/lib/libLLVM-5.0.so(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x2b)[0x7f09953839bb]
> /usr/lib/libLLVM-5.0.so(_ZN4llvm3sys17RunSignalHandlersEv+0x56)[0x7f0995381806]
> /usr/lib/libLLVM-5.0.so(+0x808953)[0x7f0995381953]
> /usr/lib/libpthread.so.0(+0x11da0)[0x7f099496cda0]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x8)[0x561fe42fc128]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
[...]
> zsh: segmentation fault (core dumped)  ldc -v  -w -d -oq -od=.dub/obj -d-version=Have_kameloso -Isource/

What can I do? Merely copying the enum into a test file and compiling with an empty main does nothing, it doesn't seem to be enough to replicate the bug.

(Arch/Manjaro 64-bit, dmd 2.077.0, ldc 1.5.0 based on 2.075.1)

[1] http://defs.ircdocs.horse/defs/numerics.html

On 21/11/2017 12:15 AM, Anonymouse wrote:
> I have a large named enum (currently 645 members) of IRC event types. It's big by neccessity[1].
> 
> I'm using dub, and both dmd and ldc successfully build it in test and debug modes, but choke and die on plain and release. I bisected it down to when I did a big addition to the enum to encompass virtually all event types there are.
> 
> dmd -v:
> [...]
>> code      common
>> function  kameloso.common.Separator.__xopEquals
>> function  kameloso.common.Separator.__xtoHash
>> function  kameloso.common.Settings.__xopEquals
>> function  kameloso.common.Settings.__xtoHash
>> function  kameloso.common.scopeguard
>> function  kameloso.common.scopeguard.scopeString
>> function  kameloso.common.scopeguard.entryString
>> function  kameloso.common.KamelosoLogger.this
>> function  kameloso.common.KamelosoLogger.writeLogMsg
>> function  kameloso.common.KamelosoLogger.beginLogMsg
>> function  kameloso.common.KamelosoLogger.logMsgPart
>> function  kameloso.common.KamelosoLogger.finishLogMsg
>> zsh: segmentation fault (core dumped)  dmd -c -v  -w -d -version=Have_kameloso -Isource/  source/arsd/dom.d
> Where it stops here varies if I comment stuff out, so I don't think the Logger is at fault.
> 
> ldc -v:
> [...]
>> code      irc
>> code      constants
>> code      connection
>> code      config
>> code      common
>> /usr/lib/libLLVM-5.0.so(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x2b)[0x7f09953839bb] 
>>
>> /usr/lib/libLLVM-5.0.so(_ZN4llvm3sys17RunSignalHandlersEv+0x56)[0x7f0995381806] 
>>
>> /usr/lib/libLLVM-5.0.so(+0x808953)[0x7f0995381953]
>> /usr/lib/libpthread.so.0(+0x11da0)[0x7f099496cda0]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x8)[0x561fe42fc128]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
>> ldc(_ZN16TemplateInstance12needsCodegenEv+0x103)[0x561fe42fc223]
> [...]
>> zsh: segmentation fault (core dumped)  ldc -v  -w -d -oq -od=.dub/obj -d-version=Have_kameloso -Isource/
> 
> What can I do? Merely copying the enum into a test file and compiling with an empty main does nothing, it doesn't seem to be enough to replicate the bug.
> 
> (Arch/Manjaro 64-bit, dmd 2.077.0, ldc 1.5.0 based on 2.075.1)
> 
> [1] http://defs.ircdocs.horse/defs/numerics.html

Source code please.

On Tuesday, 21 November 2017 at 00:15:04 UTC, Anonymouse wrote:
> I have a large named enum (currently 645 members) of IRC event types. It's big by neccessity[1].
>
> I'm using dub, and both dmd and ldc successfully build it in test and debug modes, but choke and die on plain and release. I bisected it down to when I did a big addition to the enum to encompass virtually all event types there are.
>

Try using https://github.com/CyberShadow/DustMite/wiki to obtain a minimal test case which reproduces the issue and file bug report(s).

On Tuesday, 21 November 2017 at 10:10:59 UTC, Petar Kirov [ZombineDev] wrote:
> On Tuesday, 21 November 2017 at 00:15:04 UTC, Anonymouse wrote:
>> I have a large named enum (currently 645 members) of IRC event types. It's big by neccessity[1].
>>
>> I'm using dub, and both dmd and ldc successfully build it in test and debug modes, but choke and die on plain and release. I bisected it down to when I did a big addition to the enum to encompass virtually all event types there are.
>>
>
> Try using https://github.com/CyberShadow/DustMite/wiki to obtain a minimal test case which reproduces the issue and file bug report(s).

This tool can actually be used straight from dub itself:
http://code.dlang.org/docs/commandline#dustmite

On Tuesday, 21 November 2017 at 00:36:00 UTC, rikki cattermole wrote:
> Source code please.

https://github.com/zorael/kameloso.git

I'm not proud of some of the early bits there (main.d) so don't judge me, please. I'm learning as I go.

The offending bits are IRCEvent.Type in ircstructs.d[1]. If I comment out part of it[2] making the enum smaller, it compiles. dub build always works, dub build -b plain doesn't.


[1] https://github.com/zorael/kameloso/blob/master/source/kameloso/ircstructs.d#L22
[2] https://github.com/zorael/kameloso/blob/mite/source/kameloso/ircstructs.d#L417-L700

On Tuesday, 21 November 2017 at 10:12:11 UTC, Petar Kirov [ZombineDev] wrote:
> On Tuesday, 21 November 2017 at 10:10:59 UTC, Petar Kirov [ZombineDev] wrote:
>> On Tuesday, 21 November 2017 at 00:15:04 UTC, Anonymouse wrote:
>>> I have a large named enum (currently 645 members) of IRC event types. It's big by neccessity[1].
>>>
>>> I'm using dub, and both dmd and ldc successfully build it in test and debug modes, but choke and die on plain and release. I bisected it down to when I did a big addition to the enum to encompass virtually all event types there are.
>>>
>>
>> Try using https://github.com/CyberShadow/DustMite/wiki to obtain a minimal test case which reproduces the issue and file bug report(s).
>
> This tool can actually be used straight from dub itself:
> http://code.dlang.org/docs/commandline#dustmite

I'm having trouble constructing a dustmite test of that, as I can only pass it a initial known good state by removing the parts I know exhibit the segfault.

Looking at the dmd command dub issues and testing them myself I notice that if I reorder the arguments it builds.

> $ dmd -d -oftest source/arsd/*.d source/kameloso/plugins/*.d source/kameloso/*.d
> zsh: segmentation fault (core dumped)  dmd -d -ofkek source/arsd/*.dd source/kameloso/plugins/*.d source/kameloso/*.d
>
> $ dmd -d -oftest source/kameloso/plugins/*.d source/kameloso/*.d source/arsd/*.d
> $ echo $?
> 0

On Tuesday, 21 November 2017 at 13:28:44 UTC, Anonymouse wrote:
> Looking at the dmd command dub issues and testing them myself I notice that if I reorder the arguments it builds.
>
>> $ dmd -d -oftest source/arsd/*.d source/kameloso/plugins/*.d source/kameloso/*.d
>> zsh: segmentation fault (core dumped)  dmd -d -ofkek source/arsd/*.dd source/kameloso/plugins/*.d source/kameloso/*.d
>>
>> $ dmd -d -oftest source/kameloso/plugins/*.d source/kameloso/*.d source/arsd/*.d
>> $ echo $?
>> 0

Compiling a debug dmd and running the build command in gdb, it seems to be a stack overflow at ddmd/dtemplate.d:6241, TemplateInstance::needsCodegen().

> Program received signal SIGSEGV, Segmentation fault.
> 0x00005555557ef1f7 in TemplateInstance::needsCodegen() (this=<error reading variable: Cannot access memory at address 0x7fffff7feff8>) at ddmd/dtemplate.d:6181
> (gdb) bt
> [...58165 all the way to 1...]
> #58166 0x00005555557ef351 in TemplateInstance::needsCodegen() (this=0x7fff4a8f1eb0) at ddmd/dtemplate.d:6241
> #58167 0x00005555557ef351 in TemplateInstance::needsCodegen() (this=0x7fff4a8f2640) at ddmd/dtemplate.d:6241
> #58168 0x00005555557ef351 in TemplateInstance::needsCodegen() (this=0x7fff4a8f4480) at ddmd/dtemplate.d:6241
> #58169 0x00005555557ef351 in TemplateInstance::needsCodegen() (this=0x7fff4a8f4c10) at ddmd/dtemplate.d:6241
> #58170 0x00005555557ef642 in TemplateInstance::needsCodegen() (this=0x7ffff2553280) at ddmd/dtemplate.d:6335
> #58171 0x00005555558d7851 in toObjFile::ToObjFile::visit(TemplateInstance*) (this=0x7fffffffc128, ti=0x7ffff2553280) at ddmd/toobj.d:1126
> #58172 0x00005555557f2426 in TemplateInstance::accept(Visitor*) (this=0x7ffff2553280, v=0x7fffffffc128) at ddmd/dtemplate.d:7438
> #58173 0x00005555558d59c2 in toObjFile(Dsymbol*, bool) (ds=0x7ffff2553280, multiobj=false) at ddmd/toobj.d:1335
> #58174 0x00005555558c6f19 in genObjFile(Module*, bool) (m=0x7ffff7e95bf0, multiobj=false) at ddmd/glue.d:402
> #58175 0x000055555585f7e8 in ddmd.mars.tryMain(ulong, const(char)**) (argv=0x7fffffffd928, argc=36) at ddmd/mars.d:1013
> #58176 0x000055555586003f in D main () at ddmd/mars.d:1115

This part (function final bool needsCodegen(), line 6181):

> 6233│             // Determine necessity of tinst before tnext.
> 6234│             if (tinst && tinst.needsCodegen())
> 6235│             {
> 6236│                 minst = tinst.minst; // cache result
> 6237│                 assert(minst);
> 6238│                 assert(minst.isRoot() || minst.rootImports());
> 6239│                 return true;
> 6240│             }
> 6241│---------->  if (tnext && (tnext.needsCodegen() || tnext.minst))
> 6242│             {
> 6243│                 minst = tnext.minst; // cache result
> 6244│                 assert(minst);
> 6245│                 return minst.isRoot() || minst.rootImports(); 6246│             }

On Tuesday, 21 November 2017 at 19:22:47 UTC, Anonymouse wrote:
> Compiling a debug dmd and running the build command in gdb, it seems to be a stack overflow at ddmd/dtemplate.d:6241, TemplateInstance::needsCodegen().

After a lot of trial and error I managed to find /a/ line which lets it compile under -b plain and release.

void colour(Sink, Codes...)(auto ref Sink sink, const Codes codes)
{
    // Sink is a LockingTextWriter or an Appender!string
    // Codes is a tuple of named enum members

    foreach (const code; codes)
    {
        import std.conv : to;

        if (++numCodes > 1) sink.put(';');

        sink.put((cast(size_t)code).to!string);  // <--
    }

Change size_t to uint and it compiles, keep it size_t and the compiler segfaults. Tested on two machines, both running up-to-date Arch linux, both with dmd and ldc.

The bug is too ephemeral to reduce well, if a thing like order of arguments matters.

If this is an emergent property of the rest of the program, and the size_t merely fells the house of cards, is it even worth reporting when I can't reduce it?

On Wednesday, 22 November 2017 at 15:33:46 UTC, Anonymouse wrote:
> On Tuesday, 21 November 2017 at 19:22:47 UTC, Anonymouse wrote:
>> Compiling a debug dmd and running the build command in gdb, it seems to be a stack overflow at ddmd/dtemplate.d:6241, TemplateInstance::needsCodegen().
>
> After a lot of trial and error I managed to find /a/ line which lets it compile under -b plain and release.
>
> void colour(Sink, Codes...)(auto ref Sink sink, const Codes codes)
> {
>     // Sink is a LockingTextWriter or an Appender!string
>     // Codes is a tuple of named enum members
>
>     foreach (const code; codes)
>     {
>         import std.conv : to;
>
>         if (++numCodes > 1) sink.put(';');
>
>         sink.put((cast(size_t)code).to!string);  // <--
>     }
>
> Change size_t to uint and it compiles, keep it size_t and the compiler segfaults. Tested on two machines, both running up-to-date Arch linux, both with dmd and ldc.
>
> The bug is too ephemeral to reduce well, if a thing like order of arguments matters.
>
> If this is an emergent property of the rest of the program, and the size_t merely fells the house of cards, is it even worth reporting when I can't reduce it?

You did a good investigation and I still think it's important to report it.

I managed to find a few other cases where people were having issues with needsCodegen:

https://github.com/ldc-developers/ldc/issues/2168#issuecomment-312709632
https://github.com/ldc-developers/ldc/issues/2336
https://github.com/ldc-developers/ldc/issues/2022#issuecomment-288481397
https://github.com/ldc-developers/ldc/issues/1297#issuecomment-184770787

So there's enough evidence that there's a bug somewhere around that part of the compiler and we should gather good test cases to narrow down the problem.